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March  23,  1992 

The  Honorable  David  Pryor 

Chairman,  Subcommittee  on  Federal  Services, 

Post  Office,  and  Civil  Service 
Committee  on  Governmental  Affairs 
United  States  Senate 

Dear  Mr.  Chairman: 

On  May  1 1 ,  1990,  the  Deputy  Secretary  of  Defense  testified  before  your 
Subcommittee  that  the  Department  of  Defense  (DOD)  would  not  allow 
further  procurement  of  the  Navy’s  Airborne  Self-Protection  Jammer  (ASPJ) 
program  if  aspj  did  not  meet  the  established  reliability  growth  criterion.  As 
you  requested,  we  evaluated  Don’s  compliance  with  this  commitment. 
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ASPJ  is  an  electronic  warfare  jammer  intended  to  protect  the  Navy’s  F/A-18 
and  F-14D  aircraft  from  threat  weapons.  (See  fig.  1.)  DOD  authorized  initial 
production  of  the  jammer  in  August  1989,  despite  its  marginal 
performance  during  operational  testing.  Subsequently,  the  decision  to 
initiate  ASPJ’s  Lot  I  production  contract  was  the  subject  of  hearings  before 
the  Subcommittee. 

At  the  hearing,  the  Deputy  Secretary  of  Defense  testified  that  he  had 
directed  the  establishment  of  firm  criteria  to  measure  ASPJ’s  performance 
and  reliability  and  had  directed  that  the  Defense  Acquisition  Board’  delay 
consideration  of  aspj  Lot  II  production  until  completion  of  selected 
reliability  growth  and  other  performance  tests.  The  Deputy  Secretary  also 
testified  that  DOD  would  not  allow  the  program  to  proceed  if  ASPJ  did  not 
successfully  meet  the  established  criteria. 

e 


’The  Defense  Acquisition  Board  is  a  panel  of  senior-level  DOD  officials  who  oversee  m<\jor  acquisition 
programs. 
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Results  in  Brief 


asp.j  did  not  meet  the  criterion  established  for  further  production,  non  and 
the  Navy  established  the  criterion  for  ASPJ's  reliability  growth,  and  the 
Defense  Acquisition  Board  approved  the  criterion.  However,  after  System 
failures  began  to  occur  during  the  reliability  growth  testing,  the  Navy 
changed  the  criterion  to  exclude  system  failures  attributable  to  software 
errors.  With  the  software-induced  failures  excluded,  asp.)  was  said  to  have 
met  the  reliability  growth  criterion;  and  the  Defense  Acquisition  Board, 
with  knowledge  of  the  change,  allowed  the  program  to  proceed.  If  these 
failures  had  been  included,  ASP.!  would  have  failed  the  test  by  a  large 
margin.  By  excluding  these  system  failures,  the  Navy  circumvented  POP's 
testing  standards  and  failed  to  recognize  the  adverse  impacts  of  software 
problems  experienced  with  other  electronic  warfare  systems  similar  to 
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aspj.  Moreover,  additional  reliability  growth  testing  conducted  after  the 
Defense  Acquisition  Board  allowed  the  program  to  proceed  shows  that 
ASPJ's  software  problems  are  continuing. 


Changing  Criterion 
Allowed  ASPJ  to  Pass 
Reliability  Growth 
Tests 


Defense  Acquisition 
Board  Approved 
Production  Despite 
Criterion  Change 


ASPJ's  original  reliability  growth  criterion  was  approved  by  the  Defense 
Acquisition  Board  on  November  16,  1990.  The  criterion  provided  that 
aspj  must  achieve  a  specified  measure  of  reliability  growth  called 
“mean-time-between-failure-instantaneous”-  of  at  least  75  hours  during  a 
minimum  of  650  hours  of  testing.  The  test  plan  defined  failure  to  include 
several  types  of  system  malfunctions  but  did  not  distinguish  between 
hardware-  and  software-induced  system  failures. 

ASPJ’s  reliability  growth  testing  started  in  August  1990  and  was  stopped  in 
March  1991.  The  scored  test  results  identified  17  failures  during  729  hours 
of  testing,  which  resulted  in  a  mean-time-between-failure-instantaneous  of 
83.75  hours.  According  to  the  ASPJ  Program  Office,  ASPJ  was  thus 
considered  to  have  passed  its  reliability  growth  tests. 

Not  reflected  in  the  above  results,  however,  were  43  additional  failures  of 
ASPJ’s  built-in  test  equipment  that  the  Navy  attributed  to  software 
problems.  These  failures  were  excluded  because  near  the  end  of  the  test 
period  in  March  1991,  the  Navy  revised  the  test  plan  to  provide  that 
software-induced  failures  would  not  be  relevant  to  the  scoring  of  ASPJ’s  test 
results.  If  these  failures  had  been  counted,  ASPJ’s  mean-time- 
between-failure-instantaneous  would  have  been  about  9  hours,  well  below 
the  75-hour  requirement. 


The  Defense  Acquisition  Board  approved  the  ASPJ  program  for  Lot  II 
production  after  meeting  on  June  24,  1991 .  Before  approving  Lot  II 
production,  Defense  Acquisition  Board  officials  were  provided 
documentation  revealing  that  software-induced  failures  had  been  excluded 
from  the  scoring  of  test  results. 

The  Chairman  of  the  Board  advised  us  that  the  officials  approved  Lot  II 
production  to  avoid  a  production  break,  dod  officials  also  informed  us  that 


“Mcan-timc-bctwcr::  failure-instantaneous  is  computed  using  a  formula  that  considers  total  test  time 
and  total  number  of  failures  and  that  compensates  for  progress  made  in  improving  reliability  during  the 
course  of  testing.  Thus,  a  system  showing  rapid  reliability  improvement  during  testing  would  have  a 
higher  mean-time-bctween-failure-instantaneous  than  one  that  continued  to  fail  at  a  constant  rate,  even 
if  both  had  the  same  number  of  failures  during  testing. 
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although  excluded  from  the  scoring,  the  software-induced  failures  had  not 
been  ignored  in  the  decision  to  approve  Lot  II  production.  They  said  that 
other  tests  conducted  outside  of  the  reliability  growth  test  program, 
including  developmental  flight  tests  and  contractor  tests  using  modified 
software,  had  provided  reasonable  assurance  that  the  software  problems 
had  been  corrected. 

We  disagree  that  dod  officials  had  reasonable  assurance  that  the  software 
problems  had  been  corrected.  First,  the  developmental  flight  tests  lasted 
only  45.5  hours  instead  of  the  220  hours  required  during  reliability  growth 
testing  to  verify  the  adequacy  of  software  corrections.  Moreover,  the 
purpose  of  the  flight  tests  was  not  to  evaluate  software  under  reliability 
growth  test  conditions,  but  was  to  evaluate  other  aspects  of  ASPJ’s 
performance,  such  as  its  capability  to  correctly  identify  threats.  Similarly, 
the  contractor  tests  were  also  of  insufficient  duration  to  verify  the 
adequacy  of  software  corrections.  Finally,  the  decision  memorandum 
recording  the  approval  of  Lot  II  production  directed  the  Navy  to  verify 
built-in  test  software  performance  under  actual  reliability  growth 
conditions.  This  indicates  that  DOD  officials  lacked  assurance  that  the 
software  problems  had  been  corrected. 

The  Chairman  of  the  Defense  Acquisition  Board  advised  us  that  he  was 
aware  the  system  was  not  meeting  all  its  specifications  and  had  designated 
specific  criteria  it  would  have  to  achieve  before  granting  approval  for  Lot 
III  production.  The  Chairman  also  informed  us  that  the  Deputy  Secretary  of 
Defense  lid  not  participate  in  the  decision  to  approve  ASP.l’s  Lot  II 
production. 


Excluding  Software 
Failures  Was 
Inappropriate 


By  excluding  the  43  built-in  test  failures  attributed  to  software,  the  Navy 
circumvented  non  testing  standards.  The  asp.i  test  plan  identified  Military 
Standard  2068(AS)  as  the  governing  specification  for  the  tests.  Military 
Standard  2068(AS)  classifies  built-in  test  equipment  failures  as  relevant 
failures  for  scoring  and  does  not  aut  horize  the  exclusion  of  software 
failures  from  scoring. 

In  October  1986,  Military  Standard  78 ID  superseded  2068(AS).  Military 
Standard  78  ID  does  not  exempt  built-in  test  failures  from  being  counted  as 
relevant  in  scoring  test  results.  It  does  exempt  software  failures  if  the 
failures  are  corrected  and  the  adequacy  of  the  corrections  are  verified 
during  the  reliability  growth  testing.  However,  software  changes  made  to 
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address  aspj’s  built-in  test  failures  were  not  retested  during  the  reliability 
growth  testing  preceding  the  decision  to  continue  production. 


ASPJ’s  Software  Problems  aspj's  reliability  growth  testing  resumed  after  the  approval  of  Lot  II 
Continue  production  using  the  modified  software  intended  to  correct  the  problems 

revealed  during  the  earlier  tests.  The  additional  tests  showed  that  the 
software  problems  have  not  been  resolved.  Test  records  show  that  2 1  of 
the  43  failures  that  occurred  during  the  first  phase  of  reliability  growth 
testing  recurred  during  the  second  phase  after  the  Defense  Acquisition 
Board  approved  Lot  II  production.  The  later  tests  also  revealed  additional 
software-induced  failures  not  detected  during  the  first  phase. 


Software  Problems  Degrade  Our  work  on  other  systems,  particularly  electronic  warfare  systems  similar 
Electronic  Warfare  Systems  to  ASPJ>  has  shown  that  software  problems  are  among  the  mos*  serious  in 

weapon  acquisitions.  We  have  reported  to  the  Congress  on  a  number  of 
occasions  how  software  problems  have  degraded  the  performance  of 
electronic  warfare  systems. 

In  July  1990,  we  reported  that  the  improved  ALQ- 135  jammers  produced 
for  the  Air  Force’s  F-15  aircraft  were  placed  in  storage  rather  than 
delivered  to  tactical  units  because  of  software  design  problems.3  We  also 
reported  at  that  time  that  the  Air  Force’s  ALQ-131  Block  II  for  the  F-16  and 
other  aircraft  was  being  used  by  tactical  units  in  Europe  with  an  inactive 
receiver/processor  because  of  missing  software.  The  receiver/processor  is 
a  mzyor  component  that  is  supposed  to  enhance  the  jammer’s  capability. 

In  February  1987,  we  testified  before  the  House  Committee  on  Armed 
Services  that  as  a  result  of  software  and  other  problems,  the  ALQ-161A 
defensive  avionics  system  for  the  B- 1 B  bomber  performed  poorly  and 
prevented  operation  of  a  complete  B-  IB  defensive  system.4  In  early  1991, 
non  was  still  attempting  to  complete  flight  testing  of  software  revisions  to 
achieve  adequate  performance.5 


'Electronic  Warfare:  Need  to  Strengthen  Controls  Over  Air  Force  Jammer  I’rograms 
(GAO/NSIAD-90- 1 68.  July  11,1 990)7  ' 

'The  B- 1 B  Aircraft  Program  (GAO/T-NS1AO-87-JA.  Feb.  25,  1987). 

‘Strategic  Bombers:  Issues  Related  to  the  B- 1 B  Aircraft  Program  (GAO/T-NS1AD-91-1 1,  Mar.  6,  1991). 
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Matter  for 

Congressional 

Consideration 


Because  of  the  continued  poor  performance  of  the  ASPJ,  Congress  may 
wish  to  oppose  further  production  contracts  until  operational  tests, 
scheduled  for  completion  in  fiscal  year  1992,  have  demonstrated  that  ASPJ 
will  successfully  perform  its  mission. 


Scope  and 
Methodology 


To  accomplish  our  objective,  we  examined  pertinent  testimony  given 
during  the  congressional  hearing,  test  plans  and  related  approval 
documents  setting  out  ASPJ's  reliability  growth  criterion,  summaries  of  the 
results  of  ASPJ’s  reliability  testing  and  the  related  scoring,  testing  records, 
and  documents  pertaining  to  DOD's  decision  to  continue  the  system's 
production.  We  also  examined  DOD  policy  and  test  standards  bearing  on 
the  issue  and  discussed  the  ASPJ  program  with  DOD  officials. 

We  performed  our  work  at  the  Offices  of  the  Undersecretary  of  Defense  for 
Acquisition,  Director  of  Defense  Research  and  Engineering,  Director  of 
Operational  Test  and  Evaluation,  Assistant  Secretary  of  Defense  for 
Production  and  Logistics,  and  the  Naval  Air  Systems  Command. 

We  performed  our  review  from  November  1991  to  February  1992  in 
accordance  with  generally  accepted  government  auditing  standards. 

As  requested,  we  did  not  obtain  written  agency  comments  on  this  report. 
However,  we  discussed  its  contents  with  DOD  officials  and  have 
incorporated  their  comments  where  appropriate. 


Unless  you  publicly  announce  its  contents  earlier,  we  plan  no  further 
distribution  of  this  report  until  10  days  from  its  issue  date.  At  that  time,  we 
will  send  copies  to  interested  congressional  committees;  the  Secretaries  of 
Defense  and  the  Navy;  the  Director,  Office  of  Management  and  Budget;  and 
other  interested  parties.  We  will  make  copies  available  to  others  upon 
request. 
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This  report  was  prepared  under  the  direction  of  Louis  J.  Rodrigues, 
Director,  Command,  Control,  Communications,  and  Intelligence  Issues, 
who  may  be  reached  on  (202)  275-484 1  if  you  or  your  staff  have  any 
questions  concerning  this  report.  Other  major  contributors  were  Jackie  B. 
Guin,  Assistant  Director;  Charles  A.  Ward,  Evaluator-in-Charge;  and  Peris 
Cassorla,  Evaluator. 

Sincerely  yours, 


(L--  <S*~. 


Frank  C.  Conahan 
Assistant  Comptroller  General 


(395184) 
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